{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/workshops/Idiomatic%20Programmer%20-%20handbook%202%20-%20Codelab%203.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idiomatic Programmer Code Labs\n",
    "\n",
    "## Code Labs #3 - Get Familiar with Data Curation\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Handbook 2/Part 9: Data Curation\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Preprocessing a builtin dataset (cifar-10)\n",
    "    2. Train dataset for a few epochs\n",
    "    3. Use same model architecture for larger number of classes (cifar-100)\n",
    "    4. Use image augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Let's import the builtin datasets for:\n",
    "\n",
    "    CIFAR-10: 32x32 images, 10 classes, 60000 images (6000 per class)\n",
    "    CIFAR-100: 32x32 images, 100 classes, 60000 images (600 per class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import cifar10, cifar100\n",
    "from keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Architecture\n",
    "\n",
    "Let's start with a function that will create our model using a simple CNN architecture, as follows:\n",
    "\n",
    "*Stem Group*\n",
    "\n",
    "We use the current convention of replacing a single 5x5 convolution with two 3x3 convolutions and delay downsampling to later in the network (i.e., strides=1).\n",
    "\n",
    "*Convolutional Blocks*\n",
    "\n",
    "We will use a VGG style of using sequential convolutional layers in groups, where each group doubles the number of filters, and downsample the feature maps at the end of the group using max pooling. We deviate from VGG, by using a current convention of factorizing pairs of 3x3 filters with a 3x3 and a 1x1 (bottleneck) filter.\n",
    "\n",
    "*Classifier*\n",
    "\n",
    "We will add a hidden dense layer to the classifier, and then follow it by a current convention of using a global averaging instead of flatten --this will reduce the number of parameters, where it's the current convention if the feature maps are 4x4 or greater.\n",
    "\n",
    "You fill in the blanks (replace the ??), make sure it passes the Python interpreter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import Input, Model\n",
    "from keras.layers import Conv2D, MaxPooling2D, Dropout, BatchNormalization, GlobalAveragePooling2D, Dense\n",
    "\n",
    "def convNet(input_shape, nclasses):\n",
    "    def stem(inputs, nb_filters):\n",
    "        ''' Stem Convolutional Group '''\n",
    "\n",
    "        # Use two 3x3 convolutional layers (no downsampling, strides=1)\n",
    "        x = Conv2D(nb_filters, (3, 3), strides=1, padding='same', activation='relu')(inputs)\n",
    "        x = Conv2D(nb_filters, (3, 3), strides=1, padding='same', activation='relu')(x)\n",
    "\n",
    "        # Downsample with Max Pooling\n",
    "        x = MaxPooling2D(pool_size=(2, 2), strides=2, padding='same')(x)\n",
    "        x = Dropout(0.25)(x)\n",
    "        return x\n",
    "\n",
    "    def conv_block(x, nb_filters):\n",
    "        ''' Convolutional Block '''\n",
    "\n",
    "        # A 3x3 and 1x1 factorization of two 3x3 convolutional layers\n",
    "        x = Conv2D(nb_filters, (3, 3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = Conv2D(nb_filters, (1, 1), strides=1, padding='same', activation='relu')(x)\n",
    "\n",
    "        # Downsample with Max Pooling\n",
    "        x = MaxPooling2D(pool_size=(2, 2), strides=2, padding='same')(x)\n",
    "        return x\n",
    "    \n",
    "    def classifier(x, nclasses):\n",
    "        ''' Classifier '''\n",
    "        x = GlobalAveragePooling2D()(x)\n",
    "        x = Dense(128, activation='relu')(x)\n",
    "        x = Dense(nclasses, activation='softmax')(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "    # Input and Stem Group\n",
    "    inputs = Input(input_shape)\n",
    "    x = stem(inputs, 32)\n",
    "\n",
    "    # Two Convolutional Blocks, each doubles the number of filters\n",
    "    for nb_filters in [64, 128]:\n",
    "        x = conv_block(x, nb_filters)\n",
    "\n",
    "    outputs = classifier(x, nclasses)\n",
    "\n",
    "    model = Model(inputs, outputs)\n",
    "\n",
    "    # Let's compile the model. We will use the defacto optimizer for optimization (adam)\n",
    "    # HINT: For the loss function, think categorical (multi-class) and cross entropy.\n",
    "    model.compile(loss='??', optimizer='adam', metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CIFAR-10\n",
    "\n",
    "Let's get the **Keras** builtin dataset for CIFAR-10.The image data from the `load_data()` is not preprocessed. That is, the labels are not converted to one-hot encoded and the image data is still in it's original 8-bit integer values (0..255)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Get the builtin CIFAR-10 dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# One-hot encode the labels\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "\n",
    "# Normalize the pixel data between 0 and 1\n",
    "# HINT: You are squashing the values from 0 and 255, what do you think you need to divide by?\n",
    "x_train = (x_train / ??).astype(np.float32)\n",
    "x_test  = (x_test  / ??).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Model\n",
    "\n",
    "Let's build the model for training on the CIFAR-10:\n",
    "\n",
    "    - Input is 32x32x3 (height x width x channels)\n",
    "    - 10 classes\n",
    "    \n",
    "The end of the summary output should be:\n",
    "\n",
    "```\n",
    "Total params: 140,970\n",
    "Trainable params: 140,970\n",
    "Non-trainable params: 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = convNet((32, 32, 3), 10)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model for CIFAR-10\n",
    "\n",
    "We will train the model for just a few epochs (3). Don't concern yourself about the accuracy, since we are just using a few epochs for brevity. You should get ~60% accuracy on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model using an in-memory feeding\n",
    "model.fit(x_train, y_train, batch_size=32, epochs=3, validation_split=0.1, verbose=1)\n",
    "score = model.evaluate(x_test, y_test)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CIFAR-100\n",
    "\n",
    "Let's get the **Keras** builtin dataset for CIFAR-100. This is a more challenging dataset in that it has 100 (instead of 10) classes, and fewer images per class (600 vs. 6000). \n",
    "\n",
    "In the CIFAR-10, we used normalization on the pixel data. This time, we will use standardization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the CIFAR-100 builtin dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar100.load_data()\n",
    "\n",
    "# One-hot encode the labels\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "\n",
    "# Standardize the pixel data (substract mean and divide by standard deviation of 1)\n",
    "# HINT: We are substracting the mean!\n",
    "mean = np.mean(x_train)\n",
    "std  = np.std(x_train)\n",
    "x_train = ((x_train - ??) / std).astype(np.float32)\n",
    "x_test  = ((x_test  - ??) / std).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Model\n",
    "\n",
    "Let's build the model for training on the CIFAR-100:\n",
    "\n",
    "    - Input is 32x32x3 (height x width x channels)\n",
    "    - 100 classes\n",
    "    \n",
    "The end of the summary output should be:\n",
    "\n",
    "```\n",
    "Total params: 152,580\n",
    "Trainable params: 152,580\n",
    "Non-trainable params: 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = convNet((32, 32, 3), 100)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model for CIFAR-100\n",
    "\n",
    "We will train the model for just a few epochs (3). Don't concern yourself about the accuracy, since we are just using a few epochs for brevity. You should get ~24% accuracy on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(x_train, y_train, batch_size=32, epochs=3, validation_split=0.1, verbose=1)\n",
    "score = model.evaluate(x_test, y_test)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image Augmentation with ImageDataGenerator\n",
    "\n",
    "Let's now use **Keras** ImageDataGenerator to add some image augmentation. We will do the following:\n",
    "\n",
    "    1. Vertical and Horizontal flips.\n",
    "    2. Random Rotation +/- 30 degrees.\n",
    "    \n",
    "Note, accuracy will be low (brevity) --to demonstrate using ImageDataGenerator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "# Create the generator for doing Image Augmentation\n",
    "datagen = ImageDataGenerator(vertical_flip=True, horizontal_flip=True, rotation_range=30)\n",
    "\n",
    "# For validation split, calculate the pivot point first. -i.e, where one part is training\n",
    "# and one part is validation\n",
    "pivot = int(len(x_train) * 0.9)\n",
    "\n",
    "# So for validation, it's all the data after the pivot point\n",
    "x_val = x_train[pivot:]\n",
    "y_val = y_train[pivot:]\n",
    "\n",
    "# For the remainig (before pivot point) it's the training\n",
    "# HINT: it's everything before the pivot point\n",
    "x_train = x_train[??]\n",
    "y_train = y_train[??]\n",
    "\n",
    "model = convNet((32, 32, 3), 100)\n",
    "model.fit_generator(datagen.flow(x_train, y_train, batch_size=32), epochs=3, steps_per_epoch=len(x_train)/32, \n",
    "                    validation_data=(x_val, y_val), verbose=1)\n",
    "score = model.evaluate(x_test, y_test)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Code Lab"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
